{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 问题描述\n",
    "使用tensorflow，构造并训练一个神经网络，在测试机上达到超过98%的准确率。\n",
    "#### 解题提示\n",
    "在完成过程中，需要综合运用目前学到的基础知识：\n",
    "\n",
    "深度神经网络\n",
    "\n",
    "激活函数\n",
    "\n",
    "正则化\n",
    "\n",
    "初始化\n",
    "\n",
    "并探索如下超参数设置：\n",
    "\n",
    "隐层数量\n",
    "\n",
    "各隐层中神经元数量\n",
    "\n",
    "学习率\n",
    "\n",
    "正则化因子\n",
    "\n",
    "权重初始化分布参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 初始化模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入需要的工具\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-2-4792051eef96>:3: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From /home/swing/.virtualenvs/TF/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From /home/swing/.virtualenvs/TF/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From /home/swing/.virtualenvs/TF/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting data/train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From /home/swing/.virtualenvs/TF/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting data/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From /home/swing/.virtualenvs/TF/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
     ]
    }
   ],
   "source": [
    "# 导入数据\n",
    "data_dir = 'data/'\n",
    "mnist = input_data.read_data_sets(data_dir, one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 初始化模型\n",
    "\n",
    "batch_size = 100  # 每次处理数据数\n",
    "\n",
    "n_batch = mnist.train.num_examples // batch_size # 处理完所有数据需要多少次\n",
    "\n",
    "# 学习率\n",
    "lr = 0.9\n",
    "\n",
    "# 特征占位符\n",
    "x = tf.placeholder(tf.float32, [None, 784], name='x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 添加隐层 单隐层 200个神经元就可以达到预期的效果 再多效果提升不明显\n",
    "# 默认权重初始化为标准差为0.1的数列，偏置量设为0.1  如果都设置为0的话需要的学习次数要多很多。\n",
    "w1 = tf.Variable(tf.truncated_normal([784, 200], stddev=0.1), name='w1')\n",
    "b1 = tf.Variable(tf.zeros([200]) + 0.1, name='b1')\n",
    "l1 = tf.nn.relu(tf.matmul(x, w1) + b1)   # 隐层选用relu作为激活函数 经过测试比sigmoid和tanh的效果好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 输出层  选用softmax作为激活函数\n",
    "w = tf.Variable(tf.truncated_normal([200, 10], stddev=0.1), name='w')\n",
    "b = tf.Variable(tf.zeros([10]) + 0.1, name='b')\n",
    "yp = tf.matmul(l1, w) + b  # 预测值 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "yt = tf.placeholder(tf.float32, [None, 10], name='yt')  # 真值占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-7-241008619225>:3: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See `tf.nn.softmax_cross_entropy_with_logits_v2`.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 交叉熵加上l2正则作为损失函数。 l2正则比l1正则效果要好一些。相比不加正则并没有明显的效果提升。\n",
    "cross_entropy = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(labels=yt, logits=yp)\n",
    ") + tf.contrib.layers.l2_regularizer(0.0001)(w)   # 正则参数0.0001相对比较合适"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 生成训练步骤\n",
    "train_step = tf.train.GradientDescentOptimizer(lr).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 初始化操作\n",
    "sess = tf.Session()\n",
    "init_op = tf.global_variables_initializer()\n",
    "sess.run(init_op)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:  0 lr:  0.9 accuracy:  0.9597\n",
      "epoch:  1 lr:  0.81 accuracy:  0.9677\n",
      "epoch:  2 lr:  0.7290000000000001 accuracy:  0.9707\n",
      "epoch:  3 lr:  0.6561000000000001 accuracy:  0.9704\n",
      "epoch:  4 lr:  0.5904900000000002 accuracy:  0.9751\n",
      "epoch:  5 lr:  0.5314410000000002 accuracy:  0.9779\n",
      "epoch:  6 lr:  0.47829690000000014 accuracy:  0.9751\n",
      "epoch:  7 lr:  0.43046721000000016 accuracy:  0.9784\n",
      "epoch:  8 lr:  0.38742048900000015 accuracy:  0.9784\n",
      "epoch:  9 lr:  0.34867844010000015 accuracy:  0.978\n",
      "epoch:  10 lr:  0.31381059609000017 accuracy:  0.977\n",
      "epoch:  11 lr:  0.28242953648100017 accuracy:  0.9808\n",
      "epoch:  12 lr:  0.25418658283290013 accuracy:  0.9798\n",
      "epoch:  13 lr:  0.22876792454961012 accuracy:  0.9806\n",
      "epoch:  14 lr:  0.2058911320946491 accuracy:  0.9807\n",
      "epoch:  15 lr:  0.1853020188851842 accuracy:  0.981\n",
      "epoch:  16 lr:  0.16677181699666577 accuracy:  0.9798\n",
      "epoch:  17 lr:  0.1500946352969992 accuracy:  0.9808\n",
      "epoch:  18 lr:  0.13508517176729928 accuracy:  0.9798\n",
      "epoch:  19 lr:  0.12157665459056936 accuracy:  0.9804\n",
      "epoch:  20 lr:  0.10941898913151243 accuracy:  0.9795\n",
      "epoch:  21 lr:  0.0984770902183612 accuracy:  0.9804\n",
      "epoch:  22 lr:  0.08862938119652508 accuracy:  0.981\n",
      "epoch:  23 lr:  0.07976644307687257 accuracy:  0.981\n",
      "epoch:  24 lr:  0.07178979876918531 accuracy:  0.9808\n",
      "epoch:  25 lr:  0.06461081889226679 accuracy:  0.9807\n",
      "epoch:  26 lr:  0.05814973700304011 accuracy:  0.9802\n",
      "epoch:  27 lr:  0.0523347633027361 accuracy:  0.9805\n",
      "epoch:  28 lr:  0.04710128697246249 accuracy:  0.9815\n",
      "epoch:  29 lr:  0.042391158275216244 accuracy:  0.9811\n",
      "epoch:  30 lr:  0.03815204244769462 accuracy:  0.9812\n",
      "epoch:  31 lr:  0.03433683820292516 accuracy:  0.9806\n",
      "epoch:  32 lr:  0.030903154382632643 accuracy:  0.9811\n",
      "epoch:  33 lr:  0.02781283894436938 accuracy:  0.9814\n",
      "epoch:  34 lr:  0.025031555049932444 accuracy:  0.9807\n",
      "epoch:  35 lr:  0.0225283995449392 accuracy:  0.9807\n",
      "epoch:  36 lr:  0.020275559590445278 accuracy:  0.9805\n",
      "epoch:  37 lr:  0.01824800363140075 accuracy:  0.9804\n",
      "epoch:  38 lr:  0.016423203268260675 accuracy:  0.9807\n",
      "epoch:  39 lr:  0.014780882941434608 accuracy:  0.9812\n",
      "epoch:  40 lr:  0.013302794647291147 accuracy:  0.9812\n",
      "epoch:  41 lr:  0.011972515182562033 accuracy:  0.9806\n",
      "epoch:  42 lr:  0.01077526366430583 accuracy:  0.9808\n",
      "epoch:  43 lr:  0.009697737297875247 accuracy:  0.9811\n",
      "epoch:  44 lr:  0.008727963568087723 accuracy:  0.9814\n",
      "epoch:  45 lr:  0.00785516721127895 accuracy:  0.9811\n",
      "epoch:  46 lr:  0.007069650490151055 accuracy:  0.9818\n",
      "epoch:  47 lr:  0.00636268544113595 accuracy:  0.9811\n",
      "epoch:  48 lr:  0.005726416897022355 accuracy:  0.9809\n",
      "epoch:  49 lr:  0.00515377520732012 accuracy:  0.9806\n",
      "epoch:  50 lr:  0.004638397686588107 accuracy:  0.9809\n",
      "epoch:  51 lr:  0.0041745579179292966 accuracy:  0.9809\n",
      "epoch:  52 lr:  0.003757102126136367 accuracy:  0.9808\n",
      "epoch:  53 lr:  0.00338139191352273 accuracy:  0.981\n",
      "epoch:  54 lr:  0.0030432527221704573 accuracy:  0.9808\n",
      "epoch:  55 lr:  0.0027389274499534117 accuracy:  0.9808\n",
      "epoch:  56 lr:  0.0024650347049580707 accuracy:  0.981\n",
      "epoch:  57 lr:  0.0022185312344622636 accuracy:  0.9807\n",
      "epoch:  58 lr:  0.001996678111016037 accuracy:  0.9799\n",
      "epoch:  59 lr:  0.0017970102999144335 accuracy:  0.9807\n",
      "epoch:  60 lr:  0.0016173092699229901 accuracy:  0.9804\n",
      "epoch:  61 lr:  0.0014555783429306911 accuracy:  0.9802\n",
      "epoch:  62 lr:  0.001310020508637622 accuracy:  0.9807\n",
      "epoch:  63 lr:  0.0011790184577738598 accuracy:  0.9807\n",
      "epoch:  64 lr:  0.0010611166119964739 accuracy:  0.9804\n",
      "epoch:  65 lr:  0.0009550049507968265 accuracy:  0.9813\n",
      "epoch:  66 lr:  0.0008595044557171439 accuracy:  0.9818\n",
      "epoch:  67 lr:  0.0007735540101454295 accuracy:  0.9816\n",
      "epoch:  68 lr:  0.0006961986091308866 accuracy:  0.9807\n",
      "epoch:  69 lr:  0.0006265787482177979 accuracy:  0.9809\n",
      "epoch:  70 lr:  0.0005639208733960181 accuracy:  0.9806\n",
      "epoch:  71 lr:  0.0005075287860564164 accuracy:  0.9815\n",
      "epoch:  72 lr:  0.00045677590745077476 accuracy:  0.9811\n",
      "epoch:  73 lr:  0.0004110983167056973 accuracy:  0.9806\n",
      "epoch:  74 lr:  0.0003699884850351276 accuracy:  0.9805\n",
      "epoch:  75 lr:  0.00033298963653161486 accuracy:  0.9802\n",
      "epoch:  76 lr:  0.0002996906728784534 accuracy:  0.9812\n",
      "epoch:  77 lr:  0.00026972160559060804 accuracy:  0.9805\n",
      "epoch:  78 lr:  0.00024274944503154723 accuracy:  0.9809\n",
      "epoch:  79 lr:  0.00021847450052839252 accuracy:  0.9812\n",
      "epoch:  80 lr:  0.00019662705047555326 accuracy:  0.9812\n",
      "epoch:  81 lr:  0.00017696434542799794 accuracy:  0.9807\n",
      "epoch:  82 lr:  0.00015926791088519815 accuracy:  0.981\n",
      "epoch:  83 lr:  0.00014334111979667834 accuracy:  0.9811\n",
      "epoch:  84 lr:  0.00012900700781701051 accuracy:  0.9805\n",
      "epoch:  85 lr:  0.00011610630703530947 accuracy:  0.9811\n",
      "epoch:  86 lr:  0.00010449567633177853 accuracy:  0.9808\n",
      "epoch:  87 lr:  9.404610869860067e-05 accuracy:  0.9811\n",
      "epoch:  88 lr:  8.464149782874061e-05 accuracy:  0.9815\n",
      "epoch:  89 lr:  7.617734804586655e-05 accuracy:  0.9811\n",
      "epoch:  90 lr:  6.85596132412799e-05 accuracy:  0.9812\n",
      "epoch:  91 lr:  6.170365191715192e-05 accuracy:  0.9813\n",
      "epoch:  92 lr:  5.5533286725436733e-05 accuracy:  0.9813\n",
      "epoch:  93 lr:  4.997995805289306e-05 accuracy:  0.9813\n",
      "epoch:  94 lr:  4.4981962247603756e-05 accuracy:  0.9809\n",
      "epoch:  95 lr:  4.048376602284338e-05 accuracy:  0.9814\n",
      "epoch:  96 lr:  3.6435389420559045e-05 accuracy:  0.9812\n",
      "epoch:  97 lr:  3.279185047850314e-05 accuracy:  0.9807\n",
      "epoch:  98 lr:  2.9512665430652825e-05 accuracy:  0.9805\n",
      "epoch:  99 lr:  2.6561398887587544e-05 accuracy:  0.9813\n",
      "epoch:  100 lr:  2.390525899882879e-05 accuracy:  0.9814\n"
     ]
    }
   ],
   "source": [
    "# 训练 运行100个epoch\n",
    "for epoch in range(101):\n",
    "    for batch in range(n_batch):\n",
    "        batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "        sess.run(train_step, feed_dict={x: batch_xs, yt: batch_ys})\n",
    "\n",
    "    # 每个epoch都进行准确率测试\n",
    "    correct_prediction = tf.equal(tf.argmax(yp, 1), tf.argmax(yt, 1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "    print('epoch: ', epoch, 'lr: ', lr, 'accuracy: ', sess.run(accuracy, feed_dict={x: mnist.test.images, yt: mnist.test.labels}))\n",
    "\n",
    "    lr = lr * 0.9   # 学习速率递减"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 总结\n",
    "\n",
    "本次实验在经过13个epoch之后测试的准确率就达到了98%以上，后面基本稳定在0.981左右。重复实验结果有些许波动，综合来看结果稳定在0.982左右。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 TF",
   "language": "python",
   "name": "python3 _tf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
